Repository navigation
fix(steward): recover pre-dispatch runtime failures across endpoints and stop receipt-unknown disk churn - #9159
Conversation
…and stop receipt-unknown disk churn Signed-off-by: LoopX Agent <[email protected]>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent | Gemini (Google) | self_reported
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
动机 / Motivation
受影响调用方为通过 Lark 私聊绑定 Codex 或 Antigravity(含 flexible 调度池自动切换)与管家或项目助手交互的终端用户及宿主机对账守护进程。当前版本存在两处叠加阻塞:1)当会话切换或运行在 antigravity 且发生模型调用前的准备阶段失败(如 resume_failed)时,现有 TS 恢复规划器硬编码要求 Codex 端点,且 Python 作用域探针仍检查 Codex 账号传输,导致请求永久无人恢复;2)当失败提示消息本身发出后丢失 provider 回执(provider_delivery_receipt_unknown)时,delivery_verified 无法变成 true,既拦住了上述零副作用准备失败的自动恢复,又导致对账轮询反复把该记录当作未发送正文并重复写盘覆盖同一 blocker。
修改前,绑定会话切到 antigravity 并遭遇分发前 resume_failed,且失败通知丢失回执进入 provider_delivery_receipt_unknown 时,请求永久无答复且每轮重复写入同一个 blocker;修改后,仅对已证实零副作用的预分发失败放行一次准备恢复并在冷却期退避停写磁盘。
改动后可观测结果:antigravity 与 codex 会话在分发前准备失败且通知回执未知时均能完成有限次准备恢复并投递最终答复,已落盘的 receipt-unknown 阶段不再触发重复写盘。明确非目标:不放宽任何已进入模型或工具执行阶段失败的自动重放条件,不自动重发未确认回执的历史消息。剩余未覆盖边界:真实宿主守护进程热升级与线上滞留请求的端到端实测交付需在合入部署后观测验证。
改动思路 / Approach
恢复资格裁决继续单一收敛在 TS 控制平面,辅助发现函数同域下沉到既有发现模块且不新增平行状态机。当前 PR 交付范围为预分发失败跨端点恢复、receipt-unknown 零覆写退避及 2000 行阈值内 helper 整理,线下运行态升级单独验收。
具体机制上,conversation_binding.ts 允许 planContextRecovery 在 failure_notice_receipt_unknown === true 时仅放行已证明尚未分发的准备恢复分支,并在其后第 806 行立刻以 row.delivery_verified !== true && !pendingEmpty 收口拦截所有 post-dispatch 与审阅类分支;planRuntimePreparationRecovery 扩展支持 Codex 与 Antigravity,但继续完整保留 started_at === null、upstream_turn_id === null、delta_count === 0、事件序列、60s 退避及 3 次上限。
在 Python 边界层,external_conversations.py 针对 runtime_preparation 按会话当前执行端点分流校验 Antigravity Language Server 可用性或 Codex 账号传输,并统一采用 .get("delivery_verified") is True 防止未确认回执抛出 KeyError;private_conversations.py 在 _deliver 中对已落盘同一 blocker 跳过 _atomic_write_json,并通过 private_delivery_discovery.py 将其移出 pending_body_paths、接入 pending_delivery_paths 退避队列。同期将无状态发现与历史图片校验分别整理到 file_discovery.py 与 private_delivery_discovery.py,确保两个核心文件稳定保持在 2000 行预算之内。
具体改动 / Changes
- 生产逻辑与同域抽离:更新
conversation_binding.ts(+11/-4)、external_conversations.py(+37/-105,降至 1928 行)、file_discovery.py(+91,承接_DeferredBacklogRequest/_RecoveryDiscoveryReads等发现辅助)、private_conversations.py(+44/-92,降至 1948 行)、private_delivery_discovery.py(+102/-2,承接failure_notice_receipt_unknown与只读历史图片源校验)。 - 单测与回归覆盖:在
tests/control_plane_ts/conversation_binding.test.ts(+53)与tests/test_manager_preparation_recovery.py(+75)覆盖 Antigravity 切换会话、通知回执丢失后的冷却期零写盘与满冷却期后恢复投递,并验证 post-dispatch 与关停反事实。
关键代码讲解
planContextRecovery&planRuntimePreparationRecovery:通过pendingPreparationNotice精确限定只对零副作用预分发失败豁免初始失败提示送达回执要求,并在第 806 行恢复delivery_verified === true门禁;planRuntimePreparationRecovery扩展端点范围并延续所有安全约束。_preparation_scope_available&recover_failed_request:对 Antigravity 运行期准备探测discover_antigravity_language_server,避免把gemini-*模型误传入 Codex 传输检查,同时安全读取缺失的delivery_verified布尔值。failure_notice_receipt_unknown&pending_body_paths:基于当前presentation_phase识别provider_delivery_receipt_unknown,消除重复写盘与每轮正文插队。
对主干的风险 / Risks
主要风险集中在两处边界:1)必须严格阻止 post-dispatch 失败借 failure_notice_receipt_unknown=true 重放,代码已在 conversation_binding.ts 第 806 行硬拦截并通过正反事实单测证明;2)部分已有测试中的 mock context_recovery_candidate 仅接受 1 个位置参数,private_conversations.py 仅在 receipt_unknown 为 True 时传入关键字参数,从而完全兼容存量回归套。活跃调用路径为 LarkGoalTopicRuntimeService -> LarkPrivateConversations.pending_delivery_paths / reconcile_request -> ChatExternalConversations -> planBoundConversationRequest。真实线上滞留请求的热升级自愈将在部署阶段实测核验。
我的整体评价 / Verdict
PASS:本地精确 head 623f22a28e0984bc1582e33c9764feb30dea736a 验证全部通过:node --test tests/control_plane_ts/conversation_binding.test.ts 67 passed;pytest 准备恢复与发现测试 40 passed;tests/test_lark_private_conversations.py -k "historical_image or recovery" 60 passed;tests/test_steward_interrupted_answer.py -k "not slow" 162 passed;ruff check、loopx check 与 loopx canary premerge --tier quick 均 PASS(本 Goal wait_for_ci=false,不查不等远程 CI)。
从成果影响判断:长程自治与用户体验均显著改善。UI 视觉审查不适用(已核对全部 7 个改动文件路径,仅修改 TS 控制平面策略、Core 恢复后端、Lark 私聊对账模块与单元测试,无前端或交互卡片变更)。observable_semantics 证明所有 post-dispatch 与 receipt_unknown=false 对照路径保持零行为漂移。因属作者本人提交 PR 且本次持有明确授权,按规则以 exact-head COMMENTED 评审记录上述结论。
英文简短结论
English verdict: APPROVE at exact head 623f22a; no blocking findings remain. Local TS control-plane tests (67/67), preparation/discovery tests (40/40), Lark recovery tests (60/60), interrupted-answer tests (162/162), ruff, loopx public-boundary check, and canary premerge quick tier all passed (wait_for_ci=false).
* fix(cli): preserve typed runtime diagnostics in Turn failures Signed-off-by: LoopX Agent <[email protected]> * feat(steward): support per-endpoint model preferences and Codex-to-Antigravity failover (#9141) Signed-off-by: LoopX Agent <[email protected]> * Keep the prepared Chat adapter available to a retry (#9134) A failed worker start can finish cleanup after a same-turn retry has selected the adapter. Keep its lifetime tied to the Session and let failed dispatch release its own worker registration. The regression asserts one upstream turn and explicit adapter closure with the Session. Port the fix from 4453e4bdd77486ec14b952f88ca893310b32e26c. (cherry picked from commit 173e42b4f051b885c3a3389e1cad9c5722f7b7dd) Signed-off-by: LoopX Agent <[email protected]> * fix(steward): recover zero-effect monitors and cleared-replan reports (#9143) Signed-off-by: LoopX Agent <[email protected]> * fix(steward): admit flexible_primary_available executor transitions in conversation_scope (#9145) Signed-off-by: LoopX Agent <[email protected]> * Verify quota gate scope through signed lane references (#9101) * test(quota): verify scoped gates through complete lane references Signed-off-by: LoopX Agent <[email protected]> * Test canonical quota gates in both public response modes Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * Test C1 planning and Host input from the original quota capture (#9102) * Use recorded quota decisions in the no-Todo settlement regression Signed-off-by: LoopX Agent <[email protected]> * Exercise the complete Host prompt from the captured quota source Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * Prepare the CLI budget baseline with a private Git index (#9103) Signed-off-by: LoopX Agent <[email protected]> * Exercise C1 Goal delivery and bound replan from the saved source (#9104) Signed-off-by: LoopX Agent <[email protected]> * Keep long claim overlap scopes on demand (#9086) * Link long thin-claim overlap lists to their receipts Signed-off-by: LoopX Agent <[email protected]> * Read claim overlap details without copying the lease Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * fix(quota): preserve explicit Monitor Todo selection (#9099) Carry Todo selection provenance into quota admission. Preserve explicit secondary Monitor polling and recovery from previously admitted receipts. Signed-off-by: LoopX Agent <[email protected]> * Keep ready Todo scope details on the required context read (#9106) Signed-off-by: LoopX Agent <[email protected]> * Fix archive rollback with custom install directories (#9066) * refactor(core): defer benchmark implementations during capability discovery Preserve the benchmark toolkit public exports and static types while keeping execution and experiment implementations out of metadata discovery. Qualify source and installed Core paths, and reconcile the paired minimal/Standard RFC boundary without changing activation or authority defaults. Signed-off-by: cicolord <[email protected]> * Defer Yardstick evaluation until an enabled Explore read Signed-off-by: cicolord <[email protected]> * Omit empty Todo summary lanes from ordinary Worker context Signed-off-by: cicolord <[email protected]> * fix(update): honor configured archive directories during rollback Signed-off-by: LoopX Agent <[email protected]> * fix(lark): replay recall guard on current main Signed-off-by: huangruiteng <[email protected]> * docs: record the verified LoopX 2.3.0 release baseline Signed-off-by: LoopX Agent <[email protected]> * Preserve quota spend source when prepared work is admitted Signed-off-by: LoopX Agent <[email protected]> * test(quota): qualify native saved complete-Turn checkpoints Signed-off-by: LoopX Agent <[email protected]> * Retry cleanup of an owned file mutation marker Signed-off-by: LoopX Agent <[email protected]> * Keep synthetic Git-path test data outside the public source scan Signed-off-by: LoopX Agent <[email protected]> * fix(quota): share native preparation wait with readiness Signed-off-by: LoopX Agent <[email protected]> * fix(doctor): bind package update readback to its Python environment Signed-off-by: LoopX Agent <[email protected]> * Keep Doctor discovery separate from service activation Signed-off-by: LoopX Agent <[email protected]> * Verify the recorded Python launcher before Doctor probes it Signed-off-by: LoopX Agent <[email protected]> * fix(desktop): require runtime qualification for macOS preview updates Signed-off-by: LoopX Agent <[email protected]> * fix(quota): reuse replan projection owners during recovery Signed-off-by: LoopX Agent <[email protected]> * Qualify Codex 0.169 native context replay on main Signed-off-by: LoopX Agent <[email protected]> * Stop regenerating retired quota view aliases Signed-off-by: LoopX Agent <[email protected]> * fix: pin deferred native context reads to the original capture Signed-off-by: LoopX Agent <[email protected]> * Keep native capture navigation within small output budgets Signed-off-by: LoopX Agent <[email protected]> * Retire legacy heartbeat work prompt generation Signed-off-by: LoopX Agent <[email protected]> * Capture C1 composition failures as receipt diagnostics Signed-off-by: LoopX Agent <[email protected]> * Recover interrupted native Todo readiness from the original Turn Signed-off-by: LoopX Agent <[email protected]> * Revalidate the original lease before publishing resumed native context Signed-off-by: LoopX Agent <[email protected]> * Fence native readiness publication with the canonical lease writer Signed-off-by: LoopX Agent <[email protected]> * fix(scheduler): keep Codex fallback on the original C1 runtime Signed-off-by: LoopX Agent <[email protected]> * fix(scheduler): pin the originating fallback launchers Signed-off-by: LoopX Agent <[email protected]> * Retire the duplicate Codex App scheduler projection Signed-off-by: LoopX Agent <[email protected]> * Seed the recorded scheduler profile in the elapsed-backoff fixture Signed-off-by: LoopX Agent <[email protected]> * Verify canonical cadence on refused and settled Turns Signed-off-by: LoopX Agent <[email protected]> * Pin named Core commands in saved scheduler fallbacks Signed-off-by: LoopX Agent <[email protected]> * fix(research): preserve saved case addresses when a version cannot open Signed-off-by: LoopX Agent <[email protected]> * Preserve citation focus after a saved study refresh Signed-off-by: LoopX Agent <[email protected]> * Capture authenticated LoopX receipts for Research attention (#8790) * feat(research): capture authenticated LoopX attention receipts Signed-off-by: lusendong.6789 <[email protected]> Co-authored-by: TRAE CLI <[email protected]> * fix(research): tighten attention receipt provenance Signed-off-by: lusendong.6789 <[email protected]> Co-authored-by: TRAE CLI <[email protected]> * fix(research): publish attention captures after source validation Signed-off-by: huangruiteng <[email protected]> --------- Signed-off-by: lusendong.6789 <[email protected]> Signed-off-by: huangruiteng <[email protected]> Co-authored-by: TRAE CLI <[email protected]> Co-authored-by: huangruiteng <[email protected]> * Recover the original Session and Turn from a confirmed Chat task Signed-off-by: huangruiteng <[email protected]> * perf(lark,native_chat): eliminate redundant full-directory JSON reparsing in steward chat loops (#9041) * perf(lark,native_chat): avoid full directory reparsing in steward chat loops - Make LarkOwnerAttention._route iterate capture candidates lazily so matching stops at the newest valid p2p capture. - Reuse FileChatDiscovery in PrivateIngressObserver._records while preserving strict field validation and fail-closed errors. - Reuse FileChatDiscovery in ChatExternalConversations.pending so unchanged request journals skip redundant JSON reads and stay mutation-isolated. - Add regression and scan-efficiency tests in tests/test_lark_steward_directory_scan_perf.py. Signed-off-by: LoopX Agent <[email protected]> * fix(native-chat): project small recovery metadata fields to stop data_url memory growth Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * Retire transitional smoke checks and duplicate execution (#9073) Signed-off-by: lusendong.6789 <[email protected]> Co-authored-by: TRAE CLI <[email protected]> * feat(steward): support probe-gated generic continuous_monitor todos with handoff and user actions (#9074) Signed-off-by: LoopX Agent <[email protected]> * Remove redundant defensive prose from prompts and comments (#9072) Remove redundant defensive prose while preserving published heartbeat wrappers and externally managed skill readiness. Validated with 197 related tests, independent historical wrapper migrations and negative cases, installed-skill readback, and current-main quick canary. Existing CI qualification and infrastructure limitations remain recorded in PR #9072. Signed-off-by: lusendong.6789 <[email protected]> Signed-off-by: Lusen Dong <[email protected]> * Keep original Chat retry errors visible during receipt refresh Signed-off-by: LoopX Agent <[email protected]> * fix(steward): tolerate report validator python drift and allow cancel of mid-turn failed reports (#9077) Signed-off-by: LoopX Agent <[email protected]> * feat(steward): add exact-head PR review signals and protected merge confirmation (#9081) Signed-off-by: LoopX Agent <[email protected]> * feat(steward): add isolated OpenViking long-term memory with per-speaker boundaries (#9083) Signed-off-by: LoopX Agent <[email protected]> * refactor(content-ops): slim surface module and eliminate Any (#8961) * refactor(content-ops): slim surface module and eliminate Any Signed-off-by: LoopX Agent <[email protected]> * fix(content-ops): reject malformed angle source_item_ids containers Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * fix(collaboration): batch handoff query Goal lifetime guard lock per Goal (#9085) Signed-off-by: LoopX Agent <[email protected]> * fix(hosts): raise managed service startup readiness timeout to 45s (#9087) Signed-off-by: LoopX Agent <[email protected]> * perf(manager-context): memoize history_inspect lifecycle checks and scope runtime revision in handoffs query (#9088) Eliminate the remaining O(N) Effect runtime RPCs and uncached runtime fingerprinting inside tracking.query(): enter effect_runtime_request_scope() for the query, memoize pure history_inspect lifecycle decisions across entries sharing identical Goal refs while holding the Goal lifetime guard, lazily discover legacy ingress files only when external audience recovery needs them, and preserve entry scan order across Goal batches. Signed-off-by: LoopX Agent <[email protected]> * refactor(lark): slim private_conversations below 2000 lines with net LOC reduction (#8929) Signed-off-by: LoopX Agent <[email protected]> * perf(chat,hosts): skip frontend source re-hashing on immutable installed releases and extend readiness timeout to 90s (#9089) Signed-off-by: LoopX Agent <[email protected]> * refactor(control-plane): split and slim quota monitor_poll_commit (#8885) * refactor(control-plane): split and slim quota monitor_poll_commit Signed-off-by: LoopX Agent <[email protected]> * fix(control-plane): preserve legacy lane budget and observation diagnostic order Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * refactor(quota): slim loopx/quota.py and tighten Any count to zero (#8939) Signed-off-by: LoopX Agent <[email protected]> * refactor(native-chat): slim external_conversations below 2k lines and retire ceiling (#8911) * refactor(native-chat): move steward commission mixin to steward_commands and retire external_conversations ceiling Signed-off-by: LoopX Agent <[email protected]> * fix(native-chat): respect include_raw_response in retained result recovery Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * refactor(chat): slim chat_runtime below 1,500 lines, fix entry-point project_contexts guard, and retire ceiling (#8920) * refactor(chat): move session lifecycle mixin to session_open, deduplicate runtime helpers, and retire ceiling Signed-off-by: LoopX Agent <[email protected]> * fix(chat): drop turn_done_events slot before waking waiters on Thread.start failure Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * fix(steward): paginate open PR signal GraphQL, allow signal-unavailable monitor recovery, and cache private delivery scans (#9093) Signed-off-by: LoopX Agent <[email protected]> * perf(lark,native_chat): back off noop recovery retries and cache recovery lookups (#9096) Signed-off-by: LoopX Agent <[email protected]> * perf(native_chat,lark): cache unchanged quota backlog health and interrupt observations (#9098) Signed-off-by: LoopX Agent <[email protected]> * perf(lark,native_chat): coalesce listener snapshots and cache recovery event summaries (#9100) Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: cicolord <[email protected]> Signed-off-by: LoopX Agent <[email protected]> Signed-off-by: huangruiteng <[email protected]> Signed-off-by: lusendong.6789 <[email protected]> Signed-off-by: Lusen Dong <[email protected]> Co-authored-by: cicolord <[email protected]> Co-authored-by: huangruiteng <[email protected]> Co-authored-by: Lusen Dong <[email protected]> Co-authored-by: TRAE CLI <[email protected]> * refactor(issue-fix): consolidate maintainer-correction admission and slim pr_lifecycle (#8897) Signed-off-by: LoopX Agent <[email protected]> * Fix truncated requests and responses in the Codex API relay (#8713) * fix(codex relay): fence incomplete model requests before drain Signed-off-by: LoopX Agent <[email protected]> * fix(codex relay): surface incomplete upstream responses Port the verified response framing fix to the request-admission branch. Original candidate 27bc3aee311cdfc6a00466ff9d3b58c7db36e01b. Signed-off-by: LoopX Agent <[email protected]> * Bound Codex relay writes to clients that stop reading Signed-off-by: LoopX Agent <[email protected]> * Reject upstream chunked responses missing the trailer terminator Signed-off-by: LoopX Agent <[email protected]> * Validate upstream chunk data delimiters before relay completion Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * fix(release): repair dsh closeout test, antigravity failover alias, and release download retry (#9147) Signed-off-by: LoopX Agent <[email protected]> * docs(release): update v2.4.0 release notes and dsh-loopx-plugin 0.1.1-beta.7 README (#9149) Signed-off-by: LoopX Agent <[email protected]> * fix(chat): recover saved answers and diagnose lost workers (#9120) * fix(chat): recover saved answers and diagnose lost workers Signed-off-by: LoopX Agent <[email protected]> * style(lark): keep private_conversations within module line budget Signed-off-by: LoopX Agent <[email protected]> * fix(chat): retain execution_worker_exited in pending backlog count Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * fix(chat): resume steward requests lost before dispatch (#9125) Signed-off-by: LoopX Agent <[email protected]> * fix(benchmark): restore frozen v1 archives and align checkpoint/runtime tests with C1 defaults (#9151) Signed-off-by: LoopX Agent <[email protected]> * fix(release): stabilize CLI help, architecture guards, onboarding gate, and public smokes (#9152) Signed-off-by: LoopX Agent <[email protected]> * fix(steward): recover report selection across python upgrades and prevent queue kill on observation monitors (#9155) Signed-off-by: LoopX Agent <[email protected]> * fix(release): align extensions, chat, presentation, and transport tests for v2.4.0 (#9156) Signed-off-by: LoopX Agent <[email protected]> * fix(control-plane): stabilize v2.4.0 quota, turn-envelope, and capabilities slice (Slice B1) (#9154) * fix(control-plane): stabilize v2.4.0 quota, turn-envelope, and capabilities slice Signed-off-by: LoopX Agent <[email protected]> * fix(quota): preserve tokenless identity and exact numeric types in attach_complete_turn_context Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * fix(steward): recover frozen report contract failures and rebind resumed task leases (#9157) Signed-off-by: LoopX Agent <[email protected]> * fix(steward): recover pre-dispatch runtime failures across endpoints and stop receipt-unknown disk churn (#9159) Signed-off-by: LoopX Agent <[email protected]> * fix(control_plane): align lease recovery, settlement readback, and runtime fixtures (#9158) * fix(control_plane): align lease recovery, settlement readback, and runtime fixtures Signed-off-by: LoopX Agent <[email protected]> * fix(control_plane): forward record_capture in worker outage test and harden version type guard Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * fix(control-plane): stabilize v2.4.0 explore, context, and checkpoint tail slice (Slice B4) (#9161) Signed-off-by: LoopX Agent <[email protected]> * fix(control-plane): stabilize v2.4.0 replan, monitor recovery, and staged-read slice (Slice B5) (#9160) Signed-off-by: LoopX Agent <[email protected]> * docs(release): reconcile final v2.4.0 stabilization PRs and boundary wording (#9162) * docs(release): reconcile final v2.4.0 stabilization PRs and boundary wording Signed-off-by: LoopX Agent <[email protected]> * docs(release): align #9158 and #9161 descriptions with verified commits Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * fix(steward): tolerate minor JSON delimiter glitches and recover saved report candidates (#9164) Signed-off-by: LoopX Agent <[email protected]> * fix(control-plane): preserve bound yardstick reversals in native replan openings (#9165) Signed-off-by: LoopX Agent <[email protected]> * fix(native-chat): recover loopback relay 401 failures and isolate test auth (#9167) Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> Signed-off-by: cicolord <[email protected]> Signed-off-by: huangruiteng <[email protected]> Signed-off-by: lusendong.6789 <[email protected]> Signed-off-by: Lusen Dong <[email protected]> Co-authored-by: cicolord <[email protected]> Co-authored-by: huangruiteng <[email protected]> Co-authored-by: Lusen Dong <[email protected]> Co-authored-by: TRAE CLI <[email protected]>
Problem
When a Steward or project request fails strictly before model dispatch (
resume_failed,model_auth_unavailable,execution_worker_exited_before_dispatch), two issues can strand the original private request indefinitely:antigravity(for example via flexible pool),planRuntimePreparationRecoveryhardcodedcurrent.agent_id === "codex"and_preparation_scope_availableunconditionally checked Codex native API transport.provider_delivery_receipt_unknown(started: true, noattempt),delivery_verifiednever becomestrue, soplanContextRecoveryblocked pre-dispatch preparation recovery andLarkPrivateConversations._deliver/pending_delivery_pathsrepeatedly rewroteblockerto disk on every reconcile tick.Summary
codexandantigravityinplanRuntimePreparationRecoverywhile keeping all pre-dispatch invariants (started_at === null,first_event_at === null,upstream_turn_id === null,delta_count === 0, exactturn.queued -> turn.failedsequence, 3-attempt budget, 60s exponential backoff).planContextRecoveryto enter only pre-dispatchplanRuntimePreparationRecovery/planPreparationRecoverywhenfailure_notice_receipt_unknown === true, while keepingdelivery_verified === truemandatory for all post-dispatch and review recovery paths.KeyErroron unverifieddelivery_verifiedduringrecover_failed_request, check Antigravity language-server availability forantigravityruntime preparation scope, and plumbfailure_notice_receipt_unknownfromLarkPrivateConversations._atomic_write_jsonrewrites whenblocker === "provider_delivery_receipt_unknown"is already recorded, remove already-settled receipt-unknown records frompending_body_paths, and back off non-recoverable receipt-unknown items viapending_delivery_paths/is_recovery_path.file_discovery.pyandprivate_delivery_discovery.pyto stay within module line-count ceilings.Validation
node --test tests/control_plane_ts/conversation_binding.test.ts: 67 passeduv run --extra test python -m pytest -q tests/test_manager_preparation_recovery.py tests/test_chat_runtime_preparation_recovery.py tests/test_context_recovery_discovery.py: 40 passeduv run --extra test python -m pytest -q tests/test_lark_private_conversations.py -k "historical_image or recovery": 60 passeduv run --extra test python -m pytest -q tests/test_steward_interrupted_answer.py -k "not slow": 162 passeduv run --extra test ruff checkon changed Python files: passeduv run --extra test loopx check --scan-path ...: clean (errors=0, warnings=0)uv run --extra test loopx canary premerge --from-git-diff --git-diff-base origin/main --tier quick: passed (merge_gate_passed: true,failures: 0)